The present invention relates to a man-machine interface utilizing sound data or video data (simply referred to a xe2x80x9cman-machine interfacexe2x80x9d), and in particular, to a video or information processing method and a processing apparatus for performing a process for an object with employment of sound data or video data of this object, and also to an object monitoring method and a monitoring method utilizing the processing method/apparatus.
To safely operate a large-scaled plant system such as a nuclear (atomic) power plant, an operation monitoring system including a proper man-machine interface is necessarily required. A plant is operatively maintained by way of three tasks xe2x80x9cmonitorxe2x80x9d, xe2x80x9cjudgementxe2x80x9d, and xe2x80x9cmanipulationxe2x80x9d by an operator. An operation monitoring system must be equipped with such a man-machine interface capable of smoothly achieving these three tasks by an operator. In the xe2x80x9cmonitorxe2x80x9d task, the statuses of the plant are required to be immediately, or accurately grasped. During the xe2x80x9cjudgementxe2x80x9d task, a judging material, and information to be judged must be quickly referred to an operator. During the xe2x80x9cmanipulationxe2x80x9d task, such a task environment is necessarily required in which an object to be manipulated and a result of the manipulation can be intuitively grasped, and also the manipulation intended by the operator can be quickly and correctly performed.
The man-machine interface of the conventional operation monitoring system will now be summarized with respect to each of the tasks xe2x80x9cmonitorxe2x80x9d, xe2x80x9cjudgementxe2x80x9d, and xe2x80x9cmanipulationxe2x80x9d.
(1). Monitor
Conditions within a plant may be grasped by monitoring both of data derived from various sensors for sensing pressure and temperatures and the like, and video derived from video cameras positioned at various places of the plant. Values from the various sensors are displayed on a graphic display in various ways. Also, a trend graph and a bar graph are widely utilized. On the other hand, the video derived from the video camera may be displayed on an exclusively used monitor separately provided with the graphic display. More than 40 sets of cameras are installed in a plant, which is not a rare case. While switching the cameras, and controlling the lens and directions of the cameras, an operator monitors various places in the plant. In the normal monitoring task, there is a very rare case that pictures or video derived from the cameras are observed by the operator, and it is an actual case that a utilization factor of the pictures derived from the cameras is low.
(2). Judgement
If an extraordinary case happens to occur in a plant, an operator must immediately and accurately judge what happens to occur in the plant by extensively checking a large amount of information obtained from sensors and cameras. Since the data derived from the various sensors and the pictures or video from the cameras are independently supervised or managed in the present operation monitoring system, it is difficult to reference these data and pictures by giving relationships to them, resulting in a heavy taskload on the operator.
(3). Operation
Operations are done by utilizing buttons or levers provided on an operation panel. Recently, there have been proposed such systems that an operation is performed by combining a graphic display with a touch panel, and by selecting menus and figures displayed on a screen. However, the buttons and levers provided on the operation panel, and also the menus and figures displayed on the display correspond to abstract forms irrelevant to actual objects. There is such a difficult case that an operator supposes or imagines the functions of these objects and the results of the operations. In other words, there are such problems that an operator cannot immediately understand which lever is pulled to perform a desired operation, or cannot intuitively grasp which operation command is sent to the appliance within the plant when a certain button is depressed. Also, there is another problem that since the operation panel is separately arranged with the monitor such as the camera, the bulky apparatus should be constructed.
The below-mentioned prior art has been proposed to simplify the camera switching operations and the camera remote control operations with regard to the monitoring task as described in the above item (1):
(a). Graphics produced by simulating an object to be photographed by a camera are displayed on a graphic display. A photographic place or position is instructed on the above-described graphics. In response to this instruction, the camera is remote-controlled so that a desired picture is displayed on a monitor of the camera. This type of plant operation monitoring system is known from, for instance, JP-A-61-73091.
(b). When a process device for performing either an operation, or a monitoring operation is designated by a keyboard, a process flow chart of the designated process device is graphically displayed, and simultaneously a picture of a camera for imaging the above-described process device is displayed on a screen. Such a sort of plant operation monitoring system is described in, for example, JP-A-2-224101.
(c). Based upon a designated position on a monitor screen of a camera for photographing a plant, panning, zooming and focusing operations of the camera are carried out. For instance, when an upper portion of the monitor screen is designated, the camera is panned upwardly, whereas when a lower portion of the monitor screen is designated, the camera is panned downwardly. Such a sort of plant operation monitoring system is described in, for instance, JP-A-62-2267.
On one hand, generally speaking, in a monitoring system such as a process control monitoring system, a method for visually monitoring conditions of the process has been employed by installing a monitor apparatus in a central managing room and an ITV camera (industrial television camera) at the process side and by displaying situations of the process on a monitor by way of a picture taken by this camera. This picture and sound are recorded on a recording medium such as a video tape. In an extraordinary case, the recording medium is rewound to reproduce this picture and sound.
On the other hand, data which have been sequentially sent from the process and are used as a control (control data), for instance, process data (measurement data) are displayed on either a monitor or a meter and the like of the central managing room, are stored in a database within a system, and derived from the database if an analysis is required, or an extraordinary case happens to occur. This conventional system is introduced in the plant operation history display method as opened in JP-A-60-93518.
As described above, the following problems are provided in the conventional operation monitoring systems:
(1). Since it is difficult to propagate the feeling of attendance in an actual place by way of the remote controls with employment of the keys, buttons and levers provided on the operation panel, and the menu and icon displayed on the monitor screen, the actual object to be operated and the operation result can be hardly and intuitively grasped. Thus, there are many possibilities of erroneous operations.
(2). The operator must directly switches the cameras and also directly perform the remote control operation, and cannot simply select such a camera capable of imaging a desirable scene in case that a large number of cameras are employed to monitor the scene. A cumbersome task is required to observe the desirable scene by operating the camera positioned at a remote place.
(3). There are separately provided the screen to display the picture or video derived from the video camera, the screen from which other data are referred, and the screen, or the apparatus through which the operation is instructed. Accordingly, the problems are such that the resultant apparatus becomes bulky, and the mutual reference between the video image and the other data becomes difficult.
(4). Although a video image of a camera owns a great effect to propagate the feeling of attendance, since this picture has a large quantity of information and also is not abstracted, there is a drawback that an operator can hardly and intuitively grasp a structure within the camera""s picture.
On the other hand, in accordance with a graphic representation, an important portion may be emphasized, an unnecessary portion may be simplified, and then only an essential portion may be displayed as an abstract. However, these graphic representations are separated from the actual object and the actual matter, and therefore there is a risk that an operator cannot readily imagine the relationship among the graphic representations and the actual matter/object.
(5). The video information derived from the camera is entirely, independently managed from other information (for instance, data on pressure and temperatures and the like), so that the mutual reference cannot be simply executed. As a consequence, a comprehensive judgement of the conditions can be made difficult.
On the other hand, the method opened in the above-described JP-A-61-73091 has such a merit that a desired picture can be displayed by simply designating an object to be photographed without any complex camera operations. However, an image related to the picture and control information cannot be referred to by designating a content (appliance and the like being displayed) represented in the video image. As a consequence, when an operator finds out an extraordinary portion on a monitor of a camera and tries to observe this extraordinary portion more in detail, the operator must move his eyes to the graphic screen, and must recheck the portion corresponding to the extraordinary portion on the picture with respect to the graphics.
Also, in accordance with the method described in JP-A-2-224101, there is an advantage that both of the graph representation related to the appliance designated by the keyboard and the camera image can be displayed at the same time. However, the designation of the appliance cannot be directly performed on the screen. As a consequence, when the operator finds out the extraordinary portion on the camera monitor and tries to watch this extraordinary portion more in detail, he must search the key corresponding to the extraordinary portion on the keyboard.
Moreover, in the method disclosed in JP-A-62-226786, although the operation of the camera can be designated on the screen on which the picture is being displayed without using the input device, e.g., the joystick, such a command as the pan direction, zooming-in and zooming-out of the camera is merely selected. The operator must adjust the camera how much the camera should be panned in order to more easily observe the monitoring object, which implies that this complex operation is substantially identical to that when the joystick is used. Further, since the object to be operated is limited to a single camera, the optimum picture cannot be selected from a plurality of cameras.
As described above, in the methods shown in the respective publications, the information related to the contents (graphic representations such as picture and control information) cannot be called out by directly designating the content displayed in the picture (appliances being displayed). As a result, the operator must find out the information related to the contents being represented in the picture by himself.
On the other hand, in the monitoring system such as the above-described process control monitoring system and the like, since the video information, the sound (audio) information and the process data are not mutually related with each other, when they are reproduced, or analyzed, they must be separately reproduced or analyzed in the prior art. For instance, when an extraordinary matter happens to occur, this matter is detected by the measuring device to operate the buzzer. Thereafter, the corresponding appliance is searched from the entire process diagram, and this cause and the solving method are determined, so that the necessary process is executed. In this case, to predict this cause and the failed device, a very heavy taskload is required since a large quantity of related data and pictures are needed. In the analysis with employment of the video, there are utilized the method for checking the area around the extraordinary portion based on the process data after the video is previously observed to search the area near the extraordinary portion, and the method for reproducing the picture by rewinding the video after the extraordinary point has been found out by the process data.
However, generally speaking, there are plural ITV cameras for monitoring the plant and the like. Since the pictures derived therefrom have been recorded on a plurality of videos, all of these videos must be rewound and reproduced until the desired video portion appears in order that the pictures from the respective cameras are observed with having the relationships therewith when the extraordinary matter happens to occur, and the analysis is carried out, which gives a heavy taskload to the operator.
On the other hand, it is difficult to fetch the desired data from the database, and in most case, after a large quantity of information has been printed out, the printed information is analyzed by the operations.
As described above, there are the following problems in the conventional monitoring system such as the process control monitoring system.
(1). When the video information and the audio (sound) information are reproduced, since the process data cannot be referred to at the same time, even if the information is obtained from the picture, cumbersome tasks and lengthy time are required to search the process data thereafter.
(2). Even when the process data is displayed in the trend graph or the like, and the time instant when the picture is desired to be referred to by the operator, can be recognized, both the cumbersome task and the lengthy time are required so as to display the picture. As a consequence, the actual conditions of the field cannot be quickly grasped.
(3). Even when the process data such as the extraordinary value is searched, the cumbersome task is required in order to represent the picture related to this process data.
(4). While the recorded process data is displayed, especially, when a large quantity of recorded data are displayed by the fast forwarding mode, the computer is heavily loaded.
(5). Since there is a limitation in the data display method, such demands that the contents thereof are wanted to be observed in detail, and also are wanted to be skipped, cannot be accepted. In particular, when the contents of the data are analyzed by observing them in detail, if the related picture and also sound are referred in the slow reproduction mode, more detailed analysis can be achieved. However, there is no such function.
(6). There are the operation instructions by the operator as the important element to determine the operation of the process. Since these are not reproduced, no recognition can be made whether or not the conditions of the process have been varied by effecting what sort of operation.
(7). Even when the operator remembers the executed command, since this command could not be searched, eventually prediction must be made of the time instant when the operation instruction is made by analyzing the process data and the like.
(8). As there is no relationship between the process data and the video information, even if the extraordinary matter is found out on the picture, only a skilled operator having much experience can understand what scene is imaged by this picture, and what kind of data is outputted therefrom. Accordingly, any persons who are not such a veteran could not recognize which process device has a relationship with the data.
(9). Since the place to display the video image is separated from the place to represent the process data, the operator must move his eyes and could not simultaneously watch the data and the pictures which are changed time to time.
(10). There is a problem in the reproducibility of the conventionally utilized video tape with respect to the quick access of the video data. On the other hand, if the optical disk is employed, such a quick access may be possible. However, since the video data becomes very large, a disk having a large memory capacity is required in order to record the video data.
A purpose of the present invention is to provide an information processing method and an apparatus capable of executing a process related to sound (audio) data, or video (image) data about an object based on this data.
Another purpose of the present invention is to provide a video processing method and an apparatus capable of performing a process related to a video image of at least one object displayed on a screen of display means based upon information about this object.
A further purpose of the present invention is to provide a monitoring apparatus capable of relating information for controlling a monitoring object with sound data, or video data about this monitoring object to output the related information.
To achieve such purpose, according to one aspect of the present invention, a video processing apparatus for performing a process related to a video image of at least one object displayed on a screen of a display unit, is equipped with a unit for storing information related to said object and a unit for performing a process about this object based upon the above information.
In accordance with another aspect of the present invention, an information processing apparatus for storing both of data (control data) used for controlling an object, and also data on a sound or an image related to this object, comprises a unit for relating the control data with either the sound data or the video data, and also a unit for relating the control data with the sound data or the video data based upon the relating unit to be outputted.
Preferably, an aim of the present invention is to solve the above-described problems of prior art, and to achieve at least one of the following items (1) to (6).
(1). In a remote operation monitoring system and the like, an object to be operated and an operation result can be intuitively grasped by an operator.
(2). A picture of a place to be monitored can be simply observed without cumbersome camera operations and cumbersome remote controls of cameras.
(3). The remote operation monitoring system and the like may be made compact, resulting in space saving.
(4). Merits of a camera picture and graphics are independently emphasized, and also demerits thereof may be compensated with each other.
(5). Different sorts of information can be quickly and mutually referred thereto. For instance, a temperature of a portion which is now monitored by way of a camera image can be immediately referred.
(6). A man-machine interface to achieve the above aims can be simply designed and developed.
According to the present invention, the above-described aims (1) to (5) are solved by a method having the below-mentioned steps:
(1). Object Designating Step
An object within a video image displayed on a screen is designated by employing input means such as a pointing device (will be referred to a xe2x80x9cPDxe2x80x9d). The video image is inputted from a remotely located video camera, or is reproduced from a storage medium (optical video disk, video tape recorder, disk of a computer). As the pointing device, for instance, a touch panel, a tablet, a mouse, an eyetracker, and a gesture input device and so on are utilized. Before a designation of an object, an object designatable within a picture may be clearly indicated by way of a synthesization of a graphics.
(2). Process Executing Step
Based on the object designated by the above-described object designating step, a process is executed. For example, contents of the process are as follows:
An operation command is sent by which a similar result is obtained when the designated object is operated, or has been operated. For instance, in case that the designated object corresponds to a button, such an operation instruction is sent by which a similar result can be obtained when this button is actually depressed, or has been depressed.
Based on the designated object, a picture is changed. For example, the designated object can be observed under its best condition by operating a remotely located camera. By moving a direction of a camera, a designated object is imaged at a center of a picture, and the designated object is imaged at a large size by controlling a lens. In another example, it is changed into such an image of a camera for imaging the designated object at a different angle, or into an image of a camera for photographing an object related to the designated object.
To clearly display the designated object, a graphics is synthesized with a picture and the synthesized image is displayed.
Information related to the designated object is displayed. For example, a manual, maintenance information and a structure diagram are displayed.
A list of executable process related to the designated object is displayed as a menu. A menu may be represented as a pattern (figure). In other words, several patterns are synthesized with an image to be displayed, the synthesized and displayed patterns are selected by way of PD, and then based upon the selected pattern, the subsequent process is performed.
According to the present invention, the above-described aim (1) may also be solved by a method having a step for graphically displaying a control device to control a controlled object on or near the controlled object represented in a picture.
Also, according to the present invention, the aim (2) may be solved by a method including a search key designating step for designating a search key by inputting either a text or a graphics, and a video searching step for displaying a video image in which an object matched to the search key designated by the above-described search key designating step is being represented.
In accordance with the above-identified aim (6) is solved by a method including an image display step for displaying an image inputted from a video camera, a region designation step for designating a region on the image displayed by the image display step, and a process definition step for defining a process on the region designated by the region designation step.
An object in a video picture on a screen is directly designated, and an operation instruction is sent to the designated object. While observing an actually imaged picture of the object, an operator performs an operation instruction. When the object is visually moved in response to the operation instruction, this movement is directly reflected on the picture of the camera. Thus, the operator can execute the remote operation with having such a feeling that he is actually tasking in a field by directly performing an operation with respect to the actually imaged picture. As a consequence, the operator can intuitively grasp an object to be operated and also a result of the operation, so that an erroneous operation can be reduced.
Based upon the object in the picture designated on the screen, the cameras are selected and the operation instruction is transferred to the camera. As a consequence, an image suitable for monitoring an object can be obtained by only designating the object within the image. That is to say, the operator merely designates an object desired to be observed, and thus need not select the camera but also need not remotely control the camera.
When an operation is directly given to an object within a picture, a graphics is properly synthesized therewith and the synthesized picture is displayed. For instance, once a user designates an object, such a graphic representation for clearly indicating which object has been designated is made. As a result, an operator can confirm that his intended operation is surely performed. Also in case that a plurality of processes can be executed with respect to the designated object, a menu used for selecting a desired process is displayed. This menu may be constructed by a pattern. While selecting the pattern displayed as the menu, the operator can have such a strong feeling that he actually operates the object.
Based on the object within the image designated on the screen, information is represented. As a consequence, the information related to the object within the image can be referred by only designating the object. While referring to an image and other information at the same time, it is easily possible to make a decision on conditions.
Either a text, or a pattern is inputted as a search key, and then a picture is displayed in which an object matched to the inputted search key is being displayed. The text is inputted by way of a character inputting device such as a keyboard, a speech recognition apparatus, and a handwritten character recognition apparatus. Alternatively, the pattern may be inputted by employing PD, or data which has been formed by other method is inputted. Also, the text or the pattern located in the picture may be designated as the search key. In case that the image to be search corresponds to the image from the camera, based on the search key, the camera is selected, and furthermore the direction of the camera and also the lens thereof are controlled, so that the search key can be imaged. It is also possible to clearly indicate where a portion matched to the search key is located with the picture by properly synthesizing the graphics with the image in which the object adapted to the search key is being represented. As described above, the picture is represented based on the search key, and the operator merely represents a desirable object to be seen with a language or a pattern, so that such a desirable image can be obtained for an observation purpose.
A content of a process to be executed is defined when an object within a picture has been designated by displaying the picture, designating a region on this picture, and defining a process with respect to the designated region. As a consequence, a man-machine interface for directly manipulating the object within the picture may be formed.